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How  TRAs  Got  Started 


•  “Program  managers’  ability  to  reject  immature  technologies  is 
hampered  by  (1)  untradable  requirements  that  force  acceptance 
of  technologies  despite  their  immaturity  and  (2)  reliance  on 
tools  that  fail  to  alert  the  managers  of  the  high  risks  that  would 
prompt  such  a  rejection.”  GAO/NSIAD-99-162 

•  “Identify  each  case  in  which  a  major  defense  acquisition  program  entered 
system  development  and  demonstration  ...  into  which  key  technology  has 
been  incorporated  that  does  not  meet  the  technology  maturity  requirement ... 
and  provide  a  justification  for  why  such  key  technology  was  incorporated  and 
identify  any  determination  of  technological  maturity  with  which  the  Deputy 
Under  Secretary  of  Defense  for  Science  and  Technology  did  not  concur  and 
explain  how  the  issue  has  been  resolved.”  National  Defense  Authorization 
Act  for  Fiscal  Year  2002 

•  “The  management  and  mitigation  of  technology  risk,  which  allows  less  costly 
and  less  time-consuming  systems  development,  is  a  crucial  part  of  overall 
program  management  and  is  especially  relevant  to  meeting  cost  and  schedule 
goals.  Objective  assessment  of  technology  maturity  and  risk  shall  be  a  routine 
aspect  of  DoD  acquisition.”  DoDI  5000.2,  paragraph  3.7. 2.2 

Stop  launching  programs  before  technologies  are  mature  | 


The  first  bullet  is  from  a  1999  GAO  study.  The  TRA,  which  is  a  scientific  report 
about  technology,  can’t  really  do  very  much  about  the  first  point  -untradable 
requirements-  since  that  gets  into  the  interactions  between  the  requirements 
process  and  the  acquisition  process.  But  the  TRA  is  a  tool  that  if  used  properly 
will  alert  managers  to  potential  problems  down  the  road.  The  GAO  report  also 
referred  to  TRLs  pioneered  by  NASA. 


The  second  bullet  was  a  Congressional  reaction  to  the  GAO  study.  An  annual 
report  was  called  for. 


To  ensure  that  there  was  data  for  the  annual  report,  and  of  course  to  do  the  right 
thing,  the  acquisition  regulations  were  changed  with  the  bottom  line  message. 
Don’t  start  programs  when  the  technology  is  not  ready. 
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What  is  a  TRA? 


Systematic,  metrics-based 
process  that  assesses  the 
maturity  of  Criticai 
Technoiogy  Elements  (CTEs) 

-  Uses  Technology  Readiness 
Levels  (TRLs)  as  the  metric 

Regulatory  information 
requirement  for  all 
acquisition  programs 

-  Submitted  to  DUSD(S&T)  for 
ACAT  ID  and  lAM  programs 


^  Not  a  risk  assessment 
^  Not  o  design  review 
^  Does  not  address  system 
integration 


CTEs  will  be  defined  in  the  next  slide.  TRLs  will  be  described  later  in  the 
briefing.  ACAT  ID  and  ACAT  lAM  are  the  large  defense  programs. 


While  this  slide  states  what  a  TRA  is  NOT,  the  TRA  does  contribute  to  all  of 
these. 
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Critical  Technology  Element  (CTE)  Defined 


A  technology  element  is  “critical”  if  the  system 
being  acquired  depends  on  this  technoiogy 
eiement  to  meet  operational  requirements  with 
acceptable  deveiopment  cost  and  schedule  and 

with  acceptable  production  and  operation  costs 
and  if  the  technoiogy  eiement  or  its  appiication  is 
either  new  or  novei. 

Said  another  way,  an  element  that  is  new  or  novei  or 
being  used  in  a  new  or  novel  way  is  critical  if  it  is 
necessary  to  achieve  the  successful  development 
of  a  system,  its  acquisition  or  its  operational  utility. 

CTEs  may  be  hardware,  software,  manufacturing,  or  life  cycle  related 
at  the  subsystem  or  component  level 


This  is  the  definition  in  the  TRA  Deskbook.  Key  points: 

•The  technology  does  not  have  to  enable  a  key  performance  parameter.  Any  operational 
requirement  is  OK. 

•The  technology  has  to  be  affordable  over  the  life  cycle  of  the  system. 

•Finally,  the  technology  must  be  new  or  novel.  This  does  not  imply  the  first  time  it  is  ever 
used.  Use  in  a  new  environment  is  sufficient.  This  will  be  discussed  in  more  detail  later. 


Although  CTEs  may  be  hardware,  software,  manufacturing,  or  life  cycle  related,  one  type  is  not 
treated  differently  than  another  type.  They  are  all  important.  The  only  thing  that  varies  is  what 
data  you  look  for  when  assessing  maturity. 


There  has  not  been  a  lot  of  attention  paid  to  life  cycle  related  CTEs  as  of  yet.  Only  know  of  two 
examples;  diagnostics/prognostics  on  the  F18  and  autonomous  material  handling  equipment  (an 
artificially  intelligent  forklift)  on  the  CVN  21 .  More  attention  is  needed.  Because  the  problem  is 
real. 


Suitability  is  defined  as  the  degree  to  which  a  system  can  be  placed  and  sustained  satisfactorily 
in  field  use  with  consideration  given  to  availability,  compatibility,  transportability,  interoperability, 
reliability,  wartime  usage  rates,  maintainability,  ESOH,  human  factors,  habitability,  manpower, 
logistics,  supportability,  logistics  supportability,  natural  environmental  effects  and  impacts, 
documentation  and  training  requirements.  David  Duma,  principal  deputy  director  in  OT&E,  has 
compiled  the  following  statistics.  From  1985-95  67%  of  programs  passed  OT&E  for  suitability; 
from  1995  -  1999  76%  of  programs  passed;  and  from  2000-2005  only  55%  passed. 
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Why  is  a  TRA  Important?  (1  of  2) 


•  The  Milestone  Decision  Authority 
(MDA)  uses  the  information  to  support 
a  decision  to  initiate  a  program 

-  Trying  to  apply  immature  technologies 
has  led  to  technical,  schedule,  and  cost 
problems  during  systems  acquisition 

-  TRA  established  as  a  control  to  ensure 
that  critical  technologies  are  mature, 
based  on  what  has  been  accomplished 

•  Congressional  interest 

-  MDA  must  certify  to  Congress  that 
the  technology  in  programs  has 
been  demonstrated  in  a  relevant 
environment  at  program  initiation 

-  MDA  must  justify  any  waivers  for 
national  security  to  Congress 


This  is  a  quote  from  the  testimony  of  the  Honorable  Ken  Krieg,  Undersecretary  of  Defense 
(Acquisition,  Technology,  &  Logistics)  at  a  September  27,  2005  Senate  Armed  Services 
Committee  -  “Technology  maturity  is  a  factor  in  reducing  program  risk,  thereby  reducing  near 
and  long-term  program  costs.  We  implemented  Technology  Maturity  assessments  to  assess  if 
acquisition  programs  require  more  mature  technology  before  entering  the  next  phase.  In 
addition,  we  have  increased  the  number  of  demonstrations  and  prototypes,  further  ensuring 
adequate  technology  maturity  and  military  utility  by  ‘trying  before  buying.’  ”  -  Note  that  the 
words  ‘trying  before  buying’  paraphrase  the  Packard  Commission  recommendation  to  ‘fly  before 
you  buy.’ 


Certification  required  per  Section  801  of  the  FY  2006  Defense  Authorization  Act.  Section  is 
entitled  -  requirement  for  certification  before  major  defense  acquisition  program  may  proceed  to 
Milestone  B.  Other  things  must  be  certified  as  well  -  e.g.,  affordability,  AoA  completed,  high 
likelihood  of  accomplishing  its  mission,  ... 
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Quantifying  the  Effects  of  Immature 
Technologies 

According  to  a  GAO  review  of  54  DoD  programs: 

-  Only  15%  of  programs  began  SDD  with  mature 
technology  (TRL  7) 

•  Programs  that  started  with  mature  technologies  averaged 
9%  cost  growth  and  a  7  month  schedule  delay 

•  Programs  that  did  not  have  mature  technologies  averaged 
41%  cost  growth  and  a  13  month  schedule  delay 

-  At  critical  design  review,  42%  of  programs 
demonstrated  design  stability  (90%  drawings  releasable) 

•  Design  stability  not  achievable  with  immature  technologies 

•  Programs  with  stable  designs  at  CDR  averaged  6%  cost 
growth 

•  Programs  without  stable  designs  at  CDR  averaged  46% 
cost  growth  and  a  29  month  schedule  delay 
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Source:  Defense  Acquisitions:  Assessments  of  Selected  Major  Weapon  Programs,  GAO-05-301 ,  March  2005 


This  GAO  study  was  conducted  to  show  the  effects  of  moving  forward  in  a 
program  without  having  requisite  knowledge  at  key  decision  points.  The  SDD 
decision  point  marks  program  initiation.  The  CDR  decision  point  initiates 
building  a  prototype  or  an  engineering  design  model. 


These  data  demonstrate  the  effects  of  starting  programs  with  immature 
technology.  The  top  part  of  the  chart  deals  with  technology  maturity  directly. 
The  bottom  part  of  the  chart  on  design  stability  is  also  applicable  because 
immature  technologies  inhibit  design  stability. 


Both  MDAP  and  MAIS  programs  were  included  in  the  data.  Schedule  delays 
were  measured  on  the  basis  of  Initial  Operating  Capability  (IOC)  date. 


Some  might  say  that  things  other  than  technology  immaturity  led  to  the  above 
cost  and  schedule  growth  and  that  GAO  did  not  look  at  that.  That  is  correct. 
However  there  is  no  clear  causality  between  immature  technologies  and 
inaccurate  cost  estimation,  there  is  no  clear  causality  between  immature 
technologies  and  requirement  creep,  ...  The  correlation  is  unarguable. 
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Why  is  a  TRA  Important?  (2  of  2) 


•  The  PM  uses  the  expertise  of  the  assessment 
team  and  the  rigor  and  discipline  of  the  process 
to  allow  for: 

-  Early,  in  depth  review  of  the  conceptual  product 
baseline 

-  Periodic  in-depth  reviews  of  maturation  events 

-  Highlighting  (and  in  some  cases  discover)  critical 
technologies  and  other  potential  technology  risk 
areas  that  require  management  attention  (and 
possibly  additional  resources) 

•  The  PM,  PEO,  and  CAE  use  the  results  of  the  assessment  to: 

-  Optimize  the  acquisition  strategy  and  thereby  increase  the 
probability  of  a  successful  outcome 

-  Determine  capabilities  to  be  developed  in  the  next  increment 

-  Focus  technology  investment 


In  building  the  Deskbook,  we  interviewed  a  number  of  program  managers.  This  represents  a 
summation  of  the  comments  that  we  received. 


We  recommend,  as  a  best  practice,  that  every  CTE  be  included  in  the  program’s  risk  data  base. 
In  that  way,  it’s  status  will  be  reviewed  at  each  systems  engineering  technical  review.  If  a 
technology  is  already  included  in  the  risk  data  base,  verification  by  an  independent,  well 
respected  panel  of  experts  is  important.  If  the  technology  was  not  included  in  the  risk  data  base, 
then  its  inclusion  (and  subsequent  management  actions)  can  potentially  prevent  major  problems 
down  the  line. 


The  second  bullet  reinforces  the  point  made  about  immature  technologies.  They  should  be 
deferred  to  the  next  increment  of  the  program  unless  there  are  exceptional  overriding 
circumstances. 
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This  portion  of  the  briefing  will  focus  on  the  interfaces  among  the  systems 
acquisition  process,  the  systems  engineering  process,  and  the  technology 
development  process. 
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Joint  Capabilities  Integration  and  Development 
System  (JCIDS) 


Strategic  Guidance  - 

National  Security  Strategy/National  Defense  Strategy/National  Military  Strategy 


Family  of  Joint  Future  Concepts 
Concepts  of  Operations 
Joint  Tasks 


JCIDS  governed  by  --  CJCSI  3170 


JCIDS  is  a  capabilities-based  approach  to  identifying  current  and  future  gaps.  It 
is  based  on  top-down  analyses.  The  Functional  Area  Analysis  identifies  the 
operational  tasks  to  accomplish  military  objectives.  The  Functional  Needs 
Analysis  assesses  the  ability  of  the  current  and  programmed  capabilities  to 
accomplish  the  Functional  Area  Analysis  identified  tasks.  The  result  is  a  list  of 
capability  gaps.  The  Functional  Solutions  Analysis  performs  an  operational 
based  assessment  of  potential  DOTMLPF  (doctrine,  organization,  training, 
materiel,  leadership,  personnel,  and  facilities)  approaches  to  solving  one  or 
more  of  the  existing  capability  gaps.  Not  much  weeding  out  is  done  in  the  Post- 
Independent  Analysis  unless  an  approach  is  drastically  not  feasible. 

The  final  three  boxes  in  this  chart  depict  the  interfaces  with  the  Defense 
Acquisition  System.  The  ICD  is  used  to  support  concept  refinement  decision 
and  Milestone  A  decisions  and  to  guide  the  Concept  Refinement  and  the 
Technology  Development  phases  of  the  acquisition  system.  The  CDD  supports 
a  Milestone  B  decision  by  providing  more  detail  on  the  materiel  solution  to 
provide  the  capability  previously  described  in  the  ICD.  The  CPD  is  used  to 
support  the  Milestone  C  decision  before  a  program  enters  Low  Rate  Production 
and  Operational  Test  and  Evaluation. 

This  is  a  militarily  dominated  process  with  minimal  interfaces  with  the  acquisition 
community.  No  one  in  the  technology  community  interfaces  in  the  Functional 
Solutions  Analysis.  Early  cost/performance  trades  are  not  supported  hence  the 
GAO  conclusion  that  “Program  managers’  ability  to  reject  immature  technologies 
is  hampered  by  untradable  requirements  that  force  acceptance  of  technologies 
despite  their  immaturity.” 


Overview  of  Technology  Considerations 
During  Systems  Acquisition 


This  slide  portrays  more  completely  where  the  JCIDS’  ICD,  CDD,  and  CPD 
interface  with  the  acquisition  process.  The  five  stages  of  the  acquisition 
process  are  pictured.  In  this  briefing,  we  will  pay  most  attention  to  the  first 
three  phases  since  that  is  where  the  bulk  of  the  technology  considerations 
occur. 


Say  a  few  words  about  each  phase.  Emphasize  program  initiation  at  Milestone 
B.  Technology  maturation  prior  to  program  initiation  has  been  identified  by 
GAO  as  a  commercial  best  practice.  It  also  supports  the  concept  of 
evolutionary  acquisition.  If  the  technology  is  immature,  it  should  become  the 
basis  for  future  increments  of  the  system. 

TRAs  are  required  at  Milestone  B,  Milestone  C  and  program  initiation  for  ships 
which  is  normally  at  Milestone  A. 

The  idea,  and  the  value  and  benefits,  of  conducting  TRAs  at  Milestone  A  for 
other  programs  are  currently  being  researched. 
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We  include  some  specific  do’s  and  don’ts  in  this  section  as  well  as  describe  the 
type  of  report  that  DUSD(S&T)  expects. 
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Process  Overview 


Q. 


Collect 

data 


Set  schedule 


Identify  CTEs 


Coordinate  CTEs 


Assess  CTEs;  prepare  TRA 


Coordinate  and  submit  TRA 


OSD  review 


PM  responsibility 
Coordinate  with  SAT  Exec 
Keep  bUSb(SAT)  informed 

PM  responsibility 
Best  Practice:  Independent 
review  team  appointed  by  SAT 
Exec  verifies 

PM  responsibility 
Coordinate  with  SAT  Exec 
Keep  bUSb(SAT)  informed 

SAT  Exec  responsibility 
Appoints  independent  review 
team  to  do  it;  PM  funds  it 

SAT  Exec  coordinates 
Acquisition  Executive  submits 


DUSb(SAT)  responsibility  13 


Note  that  the  process  depicted  on  this  slide  was  discussed  in  the  three  slides 
that  depicted  technology  considerations  during  the  phases  of  the  acquisition 
process.  On  this  slide,  responsibilities  are  further  clarified. 


PM  not  responsible  for  performing  the  TRA.  He  helps  identify  the  critical 
technologies  and  supports  the  assessment. 


If  backup  slide  is  NOT  being  used,  mention  that  the  schedule  should  be 
integrated  into  the  Program’s  Integrated  Master  Schedule  (IMS).  This  point  is 
made  explicitly  on  the  backup  slide. 


The  following  two  sections  of  the  briefing  deal  with  the  blocks  of  this  slide  in 
detail.  Top  3  blocks  +  collect  data  are  discussed  in  the  GTE  identification 
section.  Bottom  three  blocks  are  discussed  in  the  GTE  assessment  section. 


Independent  review  team  mentioned  twice  on  this  slide.  We  are  talking  about 
the  same  group  of  people.  The  second  time  that  it  is  mentioned,  by  assess 
GTEs,  it  is  a  requirement.  The  first  time  it  is  mentioned  by  identify  GTEs,  it  is  a 
best  practice  which  we  strongly  recommend. 
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Component  S&T  Executives 


Army 

-  Deputy  Assistant  Secretary  (Research  and 
Technology) 

Navy 

-  Chief  of  Naval  Research 
Air  Force 

-  Deputy  Assistant  Secretary  ( 
and  Engineering) 

DISA 

-  Chief  Technology  Officer 
DLA 

-  Chief  Information  Officer 
NSA 

-  Office  of  Corporate  Assessments 


Responsible  for  directing  the  TRA 
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Pictures  are  of: 

Left  side:  Edmond  Halley  (above),  Michael  Faraday  (below) 
Right  side:  Leonardo  da  Vinci  (above),  Alfred  Nobel  (below) 


Agencies  need  to  identify  who  will  perform  the  S&T  Executive  role;  often  done 
by  the  CIO  since  many  of  the  programs  are  information  technology  related. 
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Component 
SAT  Executive 
Appoints;  PM 
Funds 


independent  Review 
Team 


WBS  Elements 


Selected  from  pool  of 
recognized  experts 

-  DoD  Components 

-  FFRDCs 

-  Universities 

-  Government  agencies 

-  Industry 

-  National  Laboratories 
Final  Team  membership  based  on  work 
breakdown  structure  where  CTEs  are  located 


Manufacturing 

Sensors 

Missile  wartiing  ^ 
Communications 
‘Architecture 
^  Processing  — 
Surviwbility, 

-  Stj^ware  - 

systems" 

jy ra^ng 
Logt^ic^ 


R&M  . 

Crew  systems ' 

Antennas 

Structiuss 

PropulgiofT 

Electricatsystems 

-^'^Curjty  _  * 

^^NavigatiorC:.  ^  i, 
►  =  Safety 


Responsible  for  performing  and  preparing  the  TRA 
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Pictures  are  of  Galileo  Galilei,  Ben  Franklin,  Nicolas  Copernicus,  Isaac  Newton 
(left  to  right) 


Criteria  for  review  team  membership: 

•Real  technology  expertise  as  a  function  of  the  system  and  its  WBS 

•Knowledge  of  DoD  acquisition 

•Independent  of  program  (NOT  PM’s  CONTRACTOR) 


ODUSD(S&T)  may  suggest  that  someone  be  included  on  the  Team.  It  is  a  good 
idea  to  accommodate  such  a  request. 
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Program  Responsible  for  Scheduling  and 
Funding  the  TRA 


•  Establish  /  determine  contract  vehicle  for  funding 

-  CTE  identification 

-  Data  gathering 

-  Independent  Review  Team 

•  Training  and  preparation 

•  Assessments 

•  Report 

•  Travel 

-  Development  of  Technology  Maturation  Plans 

•  Integrate  TRA  plan  of  attack  and  milestones  into 
the  Integrated  Master  Schedule 
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Much  of  the  funding  is  for  the  independent  review  team  -  they  are  the  ones 
doing  the  assessment 


Best  practice  is  to  have  a  CLIN  in  the  contract  of  the  program  office’s 
contractors  to  support  the  TRA 
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Upper  right  hand  corner  indicates  the  portions  of  the  process  we  are  about  to 
discuss.  Note  there  are  four  blocks,  corresponding  to  the  four  rows  in  the 
timeline.  Sunsequent  slides  discuss  each  row. 


While  the  PM  has  overall  responsibility,  as  you  will  see,  others  have  key  roles. 


The  leadtimes  reflect  that  doing  the  TRA  is  no  one’s  full  time  job.  There  is  a 
great  deal  of  preparation  time  involved. 


Such  a  schedule  generally  applies  to  all  milestones.  Primarily  based  on 
experience  with  MS  B  and  MS  C  programs.  We  believe  that  MS  A  will  be 
similar.  Schedule  may  be  compressed  a  bit  to  ensure  that  there  is  enough  is 
known  to  do  a  credible  job. 


The  coordination  process  is  with  the  Component  S&T  Executive. 
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PM 

Responsible; 
Vind  Review  TearrT 
Verifies 


CTE  Identification: 


Management  Process 


Initial  review 


-  PM  led  with  program  office,  system  contractors,  and 
Govt  labs 


-  Thorough,  disciplined  and  conservative  approach 

-  Identifies  longer  list  of  candidates  to  ensure  that  no 
potential  CTE  is  overlooked 

-  Identifies  information  needed  to  determine  whether 
the  candidates  meet  the  criteria  in  the  CTE  definition 

Independent  review 

-  Conducted  by  independent  review 
team  of  experts 

-  Resolves  status  based  on  data  and 
expertise 

-  Makes  recommendations  whetheris 
candidates  meet  the  criteria 


We  distinguish  a  CTE  identification  management  process  from  a  technical 
process.  The  need  for  an  initial  review  by  the  program  office  and  the  use  of 
an  independent  review  team  are  part  of  the  management  process.  The 
details  of  what  they  do,  is  the  technical  process  which  is  described  on  the 
next  three  slides. 

This  slide  presents  a  series  of  best  practices. 

As  the  program  office  goes  through  the  possibilities,  perhaps  100  CTE 
candidates  may  be  identified.  As  data  are  gathered  and  the  criteria  are 
applied  in  the  technical  process,  most  of  the  CTE  candidates  will  be 
eliminated. 


There  is  a  backup  slide  on  the  independent  review  team.  These  are  the  points 
made  on  the  backup  slide,  to  be  made  here  if  that  slide  is  not  used. 

1 .  Component  S&T  Executive  appoints  the  independent  review  team 

2.  Criteria  for  review  team  membership:  real  technology  expertise; 
knowledge  of  DoD  acquisition;  independent  of  program  (NOT  PM’S 
CONTRACTOR) 

3.  ODUSD(S&T)  may  suggest  that  someone  be  included  on  the  Team. 
It  is  a  good  idea  to  accommodate  such  a  request. 


There  have  been  instances  where  the  independent  review  team  has  added 
CTEs  that  were  not  suggested. 
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PM 

Responsible; 
Vind  Review  TearrT 
Verifies 


_ ^TE  Identification: 

Technical  Process  (1  of  3) 


Aircraft  System 


Sys  Engineering/I 
PgmManagementI 


Peculiar 
Support  Equipmei 


Air  Vehicle  (A'f) 


System  T&E 


Training 


Industrial 

Facilities 


Comrrion  '|  ^  Airframe  §  DT&E 

Support  Equipmerjt  f  Propulsion  5  OT&E 

- Operatlonaiy  I  ^  AV  Sys  Software  §  Mock-ups 

SiteActivitation  I  f  Comm/ID  §  T&E  Support 


S  Equipment 
§  Services 
$  Facilities 


Initial  Spares  & 
Repair  Parts 


Utilize  the  work 
breakdown 
structure  (WBS) 
or  system 
architecture  for 
IT  systems,  to 
identify  CTE 
candidates  by: 

-  Establishing  the  functions  to  be  performed  by  each  system, 
subsystem,  or  component  throughout  the  WBS 

-  Determining  how  the  functions  will  be  accomplished 

-  Identifying  the  technologies  needed  to  perform  those 
functions  at  the  desired  level 
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5  Central  Computer  §  Test  Facilities 
$  Fire  Control 
§  Auto  Flight  Control 
§  Weapons  Delivery 


§  Tech  Pubs  §  Construction/ 

§  Eng  Data  Conversion/ 

5  Mgt  Data  Expansion 

5  Support  Data  5  Equipment 

5  Data  Depository  Acquisition  or 
Modernization 
5  Maintenance 
(ndusFacilitie 

Adapted  from  MIL-HDBK-881 


Need  to  think  about: 

•What  is  the  implementation  of  the  function 

•How  it  will  be  accomplished  based  on  WBS 

•What  technologies  become  associated  with  the  WBS  elements 
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PM 

Responsible; 
Vind  Review  TeairT 
Verifies 


_ ^TE  Identification: 

Technical  Process  (2  of  3) 


•  Criticality  to  the  program  criteria 

^  -  Does  the  technology  directly  impact 

I  ‘n  an  operational  requirement? 

w  -  Does  the  technology  have  a 
°  ^  significant  impact  on  an  improved 

I  I  delivery  schedule? 

K  -  Does  the  technology  have  a 
significant  impact  on  the 
affordabiiity  of  the  system? 


See  section  D.4  of  the  Deskbook  for  other  examples 
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The  CTE  does  not  have  to  relate  to  a  Key  Performance  Parameter. 


While  the  majority  of  the  questions  on  this  slide  and  the  following  few  slides 
have  a  “yes”  or  “no”  answer,  the  expectation  is  the  people  who  are  involved  in 
CTE  identification  will  also  have  an  explanation  of  their  answers.  In  some 
cases,  long  discussions  may  ensue  about  why  an  answer  is  “yes”  or  “no.” 
These  discussions  help  avoid  the  misidentification  of  a  CTE  -  either  mistakenly 
classifying  something  as  a  CTE  or  failing  to  identify  a  CTE. 

Hyperlinks  are  to  example  questions  to  uncover  more  details  in  the  CTE 
identification  process. 
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PM 

Responsible; 
Vind  Review  TearrT 
Verifies 


_ ^TE  Identification: 

Technical  Process  (3  of  3) 


I . 

H 'm 

§  SL 
«)  ’ 

°  J3 


v> 


I/I 


O  3 

“  I 


New  or  novel  criteria 

-  Is  the  technology  new  or  novel? 

-  Is  the  technology  modified? 

-  Has  the  technology  been 
repackaged  such  that  a  new 
relevant  environment  is  realized? 

-  Is  the  technology  expected  to 
operate  in  an  environment  and/or 
achieve  a  performance  beyond  its 
original  design  intention  or 
demonstrated  capability? 


Environment  key  to  “new  or  novel” 


So  now  we  have  established  that  the  GTE  candidate  is  important.  “New  or 
novel”  is  often  a  sticking  point.  For  MS  B,  the  GTE  subsystem  must  be 
demonstrated  in  a  relevant  environment.  How  you  define  environment  is 
important. 
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Environment  Examples 


Physical  Environment,  for  instance  Mechanical  _ 

Processors,  Servers  and  Electronics;  Kinetic  and^inematicj 
Thermal  and  Heat  Transfer;  Electrical  and  Electromagnetic; 
Climatic- Weather,  Temperature,  Particulate;  Network 
Infrastructure 

Logical  Environment,  fpr  instance.  Software  (Algorithm) 
Interfaces;  Security  Interfaces;  Wejr-enablement 
Data  Environment,  for  instance,  DataTortfiats  and  Databases; 
Anticipated  Data  Rates,  Data  Delay  and  Data  Throughput;  and 
Data  Packaging  and  Framing 

Security  Environment,  for  instance, -Connection  to  Firewalls; 
Security  Appli^es;  Rates  and  Methods  of  Attack  Grnfiind  Targ»t 

User  and  Use^^ironment,  for  instance.  Scalability; 
Upgradeability;  User ^H^ayior^  Adjustments;  User  Interfaces; 
Organization  Change/R^ngniihents  with  System  Impacts; 
Implem^rtSfion  Plan  ^ 


•^^•^lothers  may  be  relevant 


The  environment  where  the  technology  has  been  used  must  be  relevant  to  the 
application  of  technology  in  the  system  under  consideration.  A  radio  known  to 
work  in  a  pristine  electromagnetic  environment,  would  be  new  or  novel  in 
another  environment. 


User  and  use  environment:  Need  to  understand  who  the  users  are  and  whether 
the  technology  will  be  scalable.  Doing  effective  change  management  is  key. 
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Sample  Questions  to  Determine  if 
Environment  is  New  or  Novel 

•  Is  the  physical/logical/data  environment  in  which  this 
CTE  has  been  demonstrated  simiiarto  the  intended 
environment?  How  is  it  different?  is  the  difference 
important? 

•  Is  the  CTE  going  to  be  operating  at  or  outside  of  the 
usuai  performance  enveiope?  Do  specifications 
address  the  behavior  of  the  CTE  under  these 
conditions?  What  is  unique  or  different  about  the 
proposed  operations  environment? 

•  Do  test  data,  reports  or  anaiysis  that  compare  the 
demonstrated  environment  to  the  intended  environment 
exist?  If  modeling  and  simulation  is  an  important  aspect 
of  that  comparison,  are  the  anaiysis  techniques 
common  and  generally  accepted? 


See  Section  D.3.2  of  the  Deskbook  for  more  questions 


It  is  important  to  have  test  data,  reports  of  other  facts  to  determine  if  test 
environment  is  relevant. 


If  COTS  is  being  used  in  an  environment  other  than  the  military  environment, 
then  more  is  needed.  The  technology  should  be  considered  new  or  novel. 


For  example,  the  Sergeant  York  was  a  program  that  attempted  to  develop  a 
ground  based  air  defense  gun  system.  It  was  to  be  a  quick  program  using 
adaptations  of  existing  equipment,  especially  the  fire  control  radar  from  the  F-16. 
However,  using  the  radar  on  a  ground  vehicle  against  low  flying  aircraft  turned 
out  to  be  a  non-trivial  problem.  After  several  years  and  a  large  investment,  the 
program  was  abandoned. 
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How  Many  CTEs  Should  Be  Identified? 


•  Don’t  miss  any 

-  System  performance,  program  schedule 
and  cost  could  be  jeopardized 

•  Don’t  be  overiy  conservative 

-  If  too  many  non  critical  technologies  are 
treated  as  CTEs,  energy  and  resources 
may  be  diverted  from  the  few 
technologies  that  require  and  intensive 
maturation  effort 


If  a  disciplined  process  leads  to  an  inordinate 
number  of  CTEs,  the  proposed  development 
program  may  be  too  far  reaching 
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It  is  really  a  balancing  act.  There  is  no  arbitrary  number.  It  is  important  not  to 
miss  any. 


If  a  program  has  100  CTEs,  then  it  is  probably  too  far  reaching.  The  MDA  may 
view  this  as  a  metric  on  risk. 
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Disaggregate  CTEs  Where  Appropriate 


Software  Intensive  System  Example 


Conducted  a  thorough  analysis  to  identify  CTEs 
CTEs  included 


-  .Net  Framework  Microsoft  programming  model  and  integral 
Windows  runtime  component  for  building  XML  Web  services 
and  applications  consisting  of  the  Common  Language 
Runtime  (CLR)  and  a  unified  set  of  class  libraries 


-  COM  Callable  Wrappers  A  wrapper  around  a  .NET  object  that 
is  automatically  generated  by  the  .NET  Framework  CLR 


-  Runtime  Callable  Wrappers  A  wrapper  around  (proxy  for)  a 
COM  object  that  is  automatically  generated  by  the  .NET 
Framework  Common  Language  Runtime 


Could  have  identified  a  single  CTE  encompassing  the  three  tightly 
coupled  ones 


Maintained  disaggregation  since  each  is  new  or  novel  and  critical 
to  the  functionality  25 


It  would  not  have  been  appropriate  to  combine  the  three  into  one  CTE. 
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CTEs  May  Not  Be  Glamorous 


Ship  Example 

•  A  highly  maneuverable  load  carrying  vehicle  capable  of 
motion  in  any  direction  was  identified  as  a  CTE 

-  Intended  for  both  manual  and  autonomous  use 

•  Sensors  and  software  for  autonomous  travel 
will  be  new  as  well  as  its  use  within  the  sea 
environment 

•  This  critical  technology  provides  significant 
capability  enhancement  over  existing  material 
handling  equipment  and  supports  the  reduced 
manning  goal  of  the  ship  program 

26 


The  CTE  doesn’t  have  to  be  “the  death  ray  from  Mars.”  This  technology 
reduces  manning  on  the  ship;  it  is  a  life  cycle  related  CTE.  It  was  new  in  that 
had  never  been  done  in  the  at  sea  environment. 


CTE  Identification  Process 
Without  Designs/Performance  Criteria  (1  of  2) 

•  Situation:  Milestone  B  date  is  scheduled  prior  to 
a  fuii  definition  of  technoiogy-based 
components,  systems,  or  subsystems 

-  Contrary  to  DoDI  5000.2 

•  SDD  entrance  criteria  not  met — mature  technology 
to  meet  approved  requirements  not  established 

-  Contrary  to  good  systems  engineering  practice 

•  Program  should  be  event-driven,  not  schedule- 
driven 

•  Exit  criteria  for  the  Systems  Requirements  Review 
not  met — consistency  among  system  requirements, 
the  preferred  system  solution,  and  available 
technologies  has  not  been  achieved 

Best  practice  is  for  the  PM  to  recommend  that  the  Component 
Acquisition  Executive  defer  Milestone  B  until  the  program  is  ready 


This  situation  is  not  necessarily  one  of  immature  technologies.  Without  a 
design,  it  is  not  possible  to  do  a  good  job  in  determining  the  CTEs. 


There  is  often  a  great  deal  of  pressure  to  achieve  formal  program  initiation  on 
schedule  because  it  is  more  difficult  to  maintain  an  adequate  funding  profile 
before  program  initiation.  Frequently,  this  leads  to  a  MS  B  request  before  the 
program  has  met  the  entrance  criteria  for  SDD  despite  the  probable  effects  of 
premature  program  initiation  on  cost  and  schedule.  The  event  driven  technology 
development  process  does  not  work  well  in  a  schedule  driven  world. 


The  best  practice  is  to  defer  MS  B  until  the  program  is  ready  and  self  discipline 
by  the  programs  is  the  best  way  to  accomplish  this.  The  issue  should  be 
surfaced  by  the  PM  to  the  acquisition  executive  for  a  decision. 


If  the  technologies  are  not  clear  or  too  immature,  the  job  of  the  independent 
review  team  making  the  TRL  assessments  is  to  inform  the  PM  and  the 
Component  S&T  Executive  of  the  situation,  potentially  indicating  that  program 
initiation  should  be  deferred.  The  Milestone  Decision  Authority,  however,  can 
decide  to  press  on.  Such  a  risky  action  would  be  documented  in  the  annual 
report  to  Congress. 
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CTE  Identification  Process 
Without  Designs/Performance  Criteria  (2  of  2) 

•  If  overriding  circumstances  exist 
and  Milestone  B  is  heid  prematureiy, 
the  Component  S&T  Executive 
should  discuss  TRA  options  with 
DUSD(S&T)  as  early  as  possible 

•  One  approach  has  been 

-  Component  S&T  Executive  should  prepare  an  interim  TRA 
that  assesses  all  potential  CTEs  in  order  to 

•  Document  the  current  state  of  maturity  and  path  forward  for 
probable  CTEs 

•  Provide  an  early  indication  of  challenges  and  risks 

-  MDA  should 

•  Require  an  updated  TRA  within  3  to  6  months  in  ADM  language 

•  Give  the  program  provisional  Milestone  B  approval  pendingain 
evaluation  of  the  final  TRA 


The  key  points  are: 

•The  Components  should  provide  as  much  pertinent  information  as 
possible. 

•DUSD(S&T)  should  develop  its  recommendations  based  on  the  data 
available. 

•If  there  is  a  decision  to  proceed,  DUSD(S&T)  should  recommend  ADM 
language  requiring  another  review  in  the  near  term  when  additional  data 
are  available. 

•The  MDA  should  reserve  the  right  to  stop  the  program  at  that  time. 
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CTEs  May  Not  Be  Associated  with  a  Key 
Performance  Parameter  (1  of  2) 

Stryker  Nuclear  Biological  Chemical  Reconnaissance 
Vehicle  (NBCRV) 

-  KPPs  concerned  interoperability  and  transportability  of  the 
NBCRV  itself 

-  NBCRV  ORD  called  for  integration  of  a  standoff  chemical 
agent  detector 

•  The  mission  essential  function  is  to  detect  and 
classify 

•  The  Joint  Service  Lightweight  Standoff 
Chemical  Agent  Detector  (JSLSCAD)  is  a 
passive  infrared  detection  system  that  detects 
the  presence  or  absence  of  chemical  warfare 
agents  planned  for  the  NBCRV 

-  JSLSCAD  was  appropriately  identified  as  a  CTE 

Criticality  to  the  program  test  is:  “Does  the  technology  directly 
impact  an  operational  requirement?^'' 


The  important  thing  is  to  examine  all  of  the  operational  requirements  and  not 
limit  the  search  to  the  key  performance  parameters. 


29 


CTEs  May  Not  Be  Associated  with  a  Key 
Performance  Parameter  (2  of  2) 


Sensor  Example 

-  Two  technologies  were 
inappropriately  excluded 

•  Hyperspectral  Imagery:  New 
technology  but  not  required  to  meet 
KPPs 

*  Aided  Target  Recognition  (ATR) 
Aigorithms:  Used  to  support 
throughput  of  SAR  imagery,  not 
required  to  meet  KPPs 


Enabling  technologies  should  not  be  excluded  from  being  CTEs 


Need  to  have  these  things  to  meet  operational  requirements.  They  are  critical 
and  they  are  very  hard. 
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A  CTE  May  Be  In  Another  Program 


Ground  Vehicle  Example 

•  A  vehicle  mounted,  on  the  move,  chemical  agent  detector 
was  identified  as  a  CTE 

-  It  impacted  an  operational  requirement  and 

-  It  was  new 

•  The  proposed  solution  was  a  passive 
infrared  detection  system  that  detects  the 
presence  or  absence  of  chemical  warfare 
agents  was  an  independent  program 
initiated  in  September  1996  under  the 
Joint  Program  Office  for  Chemical, 

Biological,  Radiological,  and  Nuclear 
Defense 
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When  a  CTE  is  in  another  program,  the  program  doing  the  TRA  needs  to  assess 
the  TRL  based  on  its  own  requirements.  For  example,  the  FCS  TRA  assessed 
JTRS  with  respect  to  its  own  (FCS)  requirements.  JTRS  assessed  its  critical 
technologies  based  on  the  JTRS  requirements. 
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Modified  COTS  Product  May  Be  a  CTE 


Business  Information  Technology  System 
Example 

-  TRA  claimed  no  CTEs;  submitted  an  information  paper 
arguing  that  a  TRA  was  not  necessary 

-  Industry  leading  software  package  seiected  as  the  COTS 
basis  for  the  iT  System 

-  Software  package  being  used  in  other  Government  agencies 
as  weii  as  in  the  private  sector 

•  This  would  seem  to  imply  that  the  environment  is  not  new  or 
novel 

-  The  information  paper  went  on  to  say  “it  is  the  intent  of  DoD 
to  use  the  software  package  without  modification  to  the 
maximum  extent  possible” 

•  There  is  a  clear  basis  for  questioning  if  there  is  a  CTE.  A 
dimension  of  “new  or  novel”  is  whether  the  technology  has 
been  modified.  The  software  package  certainly  impacts  an 
operational  requirement. 


This  has  been  a  common  situation.  Programs  have  claimed  that  if  its  CTEs  are 
being  used  elsewhere  in  Government,  a  TRA  is  not  necessary  because  all  of  the 
TRLs  are  9.  All  programs  are  required  to  do  a  TRA. 
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CTE  Coordination  and 
Data  Collection 


PM  submits  list  to  the  Component  S&T 
Executive  and  requests  a  TRA 

S&T  Executive  may  add  CTEs  if  it  is  felt 
that  special  attention  is  warranted 

PM  collects  evidence  of  CTE  maturity 

-  Ongoing  process  throughout  CTE 
identification 

-  May  include 

•  Component  and  subsystem  test  descriptions 

•  Anaiyses 

•  Environments 


•  Results 


Keep  DUSD(S&T)  informed;  may  suggest  additional  CTEs 
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Remember,  the  Component  S&T  Executive  (or  his/her  designated  agent)  is  the 
one  who  coordinates  on  the  CTE  list. 


Either  the  PM  or  the  Component  S&T  Executive  may  add  CTEs  over  and  above 
those  recommended  by  the  independent  review  team,  if  they  feel  that  special 
attention  is  warranted.  ODUSD(S&T)  may  suggest  additional  CTEs,  it  is  a  good 
idea  to  accommodate  such  a  request. 
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Independent  Review  Team 


Information  Needs  (1  of  3) 


*  Program  overview  to  set  the  foundation  for  the  CTE 
assessments 

-  Concept  of  operations 

-  Program  master  schedule 

•  Identify  significant  milestones,  items  on  the  critical  path  and  status 
progress 

-  Operational  performance  requirements 

•  Highlight  KPPs,  in  general,  and  those  operational  requirements  that 
will  be  directly  influenced  by  the  CTEs  to  be  assessed 

-  Challenges  associated  with  the  CTEs  to  be  assessed 

-  Technology  maturation  roadmap 

•  Highlight  those  maturation  events  that  have  been  accomplished  and 
those  yet  to  occur 

-  Overall  system  architecture 

•  Highlights  the  CTE  system  /  subsystem  elements  that  will  be 
assessed 


This  reflects  things  that  the  PM  needs  to  tell  the  independent  review  team. 

The  independent  review  team  needs  a  good  handle  on  the  operational 
requirements,  what  the  PM  thinks  the  challenges  are  with  the  CTEs,  and  the 
events  that  will  occur  to  demonstrate  how  the  CTEs  will  mature. 
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Independent  Review  Team 


Information  Needs  (2  of  3) 


*  Introduction  to  the  subsystems  containing  the  CTEs 

-  Technical  description  of  the  subsystem,  to  include  physical 
architecture,  highlighting  CTEs  (components  and/or  packaging), 
explaining  why  other  technologies  within  subsystem  are  non- 
critical,  and  differentiating  subsystem  and  elements  from  that  of 
potentially  similar  designs  (i.e.,  highlight  any  uniqueness) 

-  Description  of  the  subsystem’s  intended  function  in  the  design 

*  Significance  of  the  CTEs  relative  to  the  subsystem 

*  Significance  of  the  subsystem  relative  to  the  system  overall  design 

*  Traceability  of  the  subsystem  relative  to  the  applicable  operational 
requirements  and  state  whether  impact  to  a  KPP 

-  Schedule  for  the  design  and  integration  of  the  subsystem,  clearly 
identifying  critical  path  events  and,  if  relevant, 
expectation/deliveries  from  suppliers 

-  Block  diagram  and  risk  assessment  for  the  subsytem 

-  Roadmap  of  on-going  and  planned  maturation  activities  and  h(^ 

_ these  events  can  influence  the  master  design  schedule _ 

This  represents  a  data  intensive  description  of  the  components  of  WBS  or 
architecture. 
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Independent  Review  Team 


Information  Needs  (3  of  3) 


•  Status  of  CTE 

-  Accomplishments  that  directly  reflect  CTE  maturation 

•  Use  TRL  rating  factors  as  a  guideline 

•  State  quantitative  facts  where  possible  in  order  to  temper  and 
legitimize  significance  of  the  technology  maturation 
accomplishments 

•  Describe  the  measurement  environment  and  methodology  used 

•  Identify  who  witnessed  the  subsystem/technology  maturation 
accomplishments 

-  Tangible  evidence  of  CTE  maturation  accomplishments  (e.g., 
hardware,  pictures,  displays,  technical  papers,  reports,  etc.) 

•  Clearly  state  what  is  and  is  not  represented  by  the  evidence 

-  Relevant  CTE  maturation  leveraged  from  other  programs 

•  Clearly  state  any  differences  between  this  program  and  the 
leveraged  program  to  appreciate  significance  of  maturation  events 

-  Significant  maturation  events  that  fall  short  or  have  not  been  36 
accomplished 


The  independent  review  team  will  be  looking  for  data  and  graphs  to  support  the 
assessment  in  different  environments. 

The  maturation  process  is  not  completely  smooth.  There  will  be  interest  in 
events  where  the  technology  did  not  perform  as  expected  or  desired. 
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Upper  right  hand  corner  indicates  the  portions  of  the  process  we  are  about  to 
discuss.  Note  there  are  three  blocks,  corresponding  to  the  top  three  major  rows 
in  the  timeline.  Subsequent  slides  discuss  each  row.  Last  row  is  just  the 
milestone  review  itself. 


While  the  Component  S&T  has  overall  responsibility  for  the  top  two  rows,  as  you 
will  see,  others  have  key  roles. 


Such  a  schedule  generally  applies  to  all  milestones.  Key  is  to  get  the  results  to 
DUSD(S&T)  early  enough  for  the  review  and  evaluation. 
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TRL  Overview 


•  Measures  technology  maturity 

•  Indicates  what  has  been  accomplished  in  the 
development  of  a  technology 

-  Theory,  laboratory,  field 

-  Relevant  environment,  operational 
environment 

-  Subscale,  full  scale 

-  Breadboard,  brassboard,  prototype 

-  Reduced  performance,  full 
performance 

•  Does  not  indicate  that  the  technology  is  right  for 
the  job  or  that  application  of  the  technology  will 
result  in  successful  development  of  the  system 


Concept  was  pioneered  by  NASA. 


The  TRL  is  not  a  predictor  of  where  you  are  going  to  be.  It  does  not  comment 
on  whether  SE  has  picked  the  right  design  or  the  right  technologies. 
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Hardware  TRLs 


1.  Basic  principles  observed  and  reported 

2.  Technology  concept  and/or  application 
formulated 

3.  Analytical  and  experimental  critical 
function  and/or  characteristic  proof  of 
concept 

4.  Component  and/or  breadboard 
validation  in  a  laboratory  environment 

5.  Component  and/or  breadboard 
validation  in  a  relevant  environment 

6.  System/subsystem  model  or  prototype 
demonstration  in  a  relevant  environment 

7.  System  prototype  demonstration  in  an 
operational  environment 

8.  Actual  system  completed  and  qualified 
through  test  and  demonstration 

9.  Actual  system  proven  through  sg 

successful  mission  operations 


TRLs  1  thru  3  are  in  the  tech  base,  pre  MS  A. 


TRL  4  or  greater  is  the  preferred  maturity  at  MS  A. 
TRL  6  is  required  for  MS  B. 

Technologies  should  be  TRL  7  or  greater  at  MS  C. 


39 


TRL  4  Hardware 

Minimum  Maturity  at  Milestone  A 


•  Definition:  Component  and/or  breadboard  validation  in  a 
laboratory  environment. 

•  Description:  Basic  technological  components  are  integrated 
to  establish  that  they  will  work  together.  This  is  relatively 
“low  fidelity”  compared  with  the  eventual  system.  Examples 
include  integration  of  “ad  hoc”  hardware  in  the  laboratory. 

•  Supporting  Information:  System 
concepts  that  have  been  considered 
and  results  from  testing  laboratory- 
scale  breadboard(s).  References 
who  did  this  work  and  when. 

Provide  an  estimate  of  how 
breadboard  hardware  and  test 
results  differ  from  the  expected 
system  goals. 


This  is  relatively  low  fidelity,  based  on  lab  tests  or  bench  tests,  and  then  your 
estimation  of  how  the  system  is  going  to  work.  However,  it  is  a  real  scientific 
statement  of  what  has  been  accomplished  at  the  bread 
board/component/laboratory  environment. 
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TRL  5  Hardware 


•  Definition:  Component  and/or  breadboard 
validation  in  a  relevant  environment. 

•  Description:  Fidelity  of  breadboard 
technology  increases  significantly.  The  basic 
technological  components  are  integrated 
with  reasonably  realistic  supporting 
elements  so  they  can  be  tested  in  a 
simulated  environment.  Examples  include 
“high-fidelity”  laboratory  integration  of 
components. 

•  Supporting  information:  Resuits  from  testing  a  iaboratory 
breadboard  system  are  integrated  with  other  supporting  elements 
in  a  simulated  operational  environment.  How  does  the  “relevant 
environment”  differ  from  the  expected  operational  environment? 
How  do  the  test  resuits  compare  with  expectations?  What 
problems,  if  any,  were  encountered?  Was  the  breadboard  system 
refined  to  more  nearly  match  the  expected  system  goals? 


A  relevant  environment  example  may  be  the  electromagnetic  environment  or 
conditions. 
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TRL  6  Hardware 

Minimum  Maturity  at  Milestone  B 

•  Definition:  System/subsystem  modei  or  prototype  demonstration  in 
a  reievant  environment. 

•  Description:  Representative  modei  or  prototype  system,  which  is 
weli  beyond  that  of  TRL  5,  is  tested  in  a  reievant  environment. 
Represents  a  major  step  up  in  a  technology’s  demonstrated 
readiness.  Examples  include  testing  a  prototype  in  a  high-fidelity 
laboratory  environment  or  in  a  simulated  operational  environment. 

•  Supporting  Information:  Results  from  laboratory 
testing  of  a  prototype  system  that  is  near  the 
desired  configuration  in  terms  of  performance, 
weight,  and  volume.  How  did  the  test  environment 
differ  from  the  operational  environment?  Who 
performed  the  tests?  How  did  the  test  compare 
with  expectations?  What  problems,  if  any,  were 
encountered?  What  are/were  the  plans,  options, 
or  actions  to  resolve  problems  before  moving  |p 
the  next  level? 


The  level  of  maturity  is  achieved  after  you  have  tested  the  system  or  subsystem 
in  a  relevant  environment.  You  should  be  testing  something  very  close  to  its 
final  configuration. 


42 


TRL  7  Hardware 

Minimum  Maturity  at  Milestone  C 

•  Definition:  System  prototype  demonstration  in  an  operationai 
environment. 

•  Description:  Prototype  near  or  at  pianned  operational  system. 
Represents  a  major  step  up  from  TRL  6  by  requiring 
demonstration  of  an  actual  system  prototype  in  an  operational 
environment  (e.g.,  in  an  aircraft,  in  a  vehicle,  or  in  space). 
Examples  include  testing  the  prototype  in  a  test  bed  aircraft. 

•  Supporting  Information:  Results 
from  testing  a  prototype  system 
in  an  operationai  environment. 

Who  performed  the  tests?  How 
did  the  test  compare  with 
expectations?  What  probiems,  if 
any,  were  encountered?  What 
are/were  the  plans,  options,  or 
actions  to  resoive  probiems 
before  moving  to  the  next  level? 


TRL  7  is  the  preferred  maturity  at  MS  B,  based  on  GAO’s  review  of  industry  best 
practices. 
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Software  TRLs 


1. 

2. 

3. 

4. 


5. 


6. 


7. 


8. 


9. 


Basic  principles  observed  and  reported. 

Technology  concept  and/or  application  formulated. 
Analytical  and  experimental  critical  function  and/or 
characteristic  proof  of  concept 
Module  and/or  subsystem  validation  in  a 
laboratory  environment,  i.e.  software  prototype 
development  environment 

Module  and/or  subsystem  validation  in  a  relevant 
environment  a/iL 

Module  and/or  subsystem  validation  in  a  relevant 
end-to-end  environment 

System  prototype  demonstration  in  an  operational 
high  fidelity  environment 
Actual  system  completed  and  mission  qualified 
through  test  and  demonstration  in  an  operational 
environment 

Actual  system  proven  through  successful  mission 
proven  operational  capabilities 
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Hardware  technology  may  include  software  that  executes  on  the  hardware  if  (1) 
the  software  is  not  being  developed  or  modified  as  part  of  the  acquisition  or  (2) 
the  software  is  not  the  reason  for  placing  the  element  on  the  CTE  list.  Therefore 
a  tactical  subsystem  with  imbedded  software  may  be  identified  as  a  CTE  and  its 
maturity  may  be  assessed  using  only  the  hardware  TRLs. 


Generally,  the  software  TRLs  are  analogous  to  the  hardware  TRLs.  The 
terminology  changes.  The  word  “module  and/or  subsystem”  replaces 
“component  and/or  bread  board.” 


Note:  End-to-end  environment  includes  the  data  and  the  sequencing  of 
transactions  in  that  environment. 
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TRL  4  Software 

Minimum  Maturity  at  Milestone  A 

•  Definition:  Module  and/or  subsystem  vaiidation  in 
a  laboratory  environment  (i.e.,  software  prototype 
development  environment). 

•  Description:  Basic  software  components  are 
integrated  to  establish  that  they  will  work  together. 

They  are  relatively  primitive  with  regard  to 
efficiency  and  robustness  compared  with  the 
eventual  system.  Architecture  development 
initiated  to  include  interoperability,  reliability, 
maintainability,  extensibility,  scalability,  and 
security  issues.  Emulation  with  current/legacy 
elements  as  appropriate.  Prototypes  developed  to 
demonstrate  different  aspects  of  eventual  system. 

•  Supporting  Information:  Advanced  technology  development, 
stand-alone  prototype  solving  a  synthetic  full-scale  problem,  or 
standalone  prototype  processing  fully  representative  data  sets. 


The  software  prototype  is  analogous  to  the  bench  test  in  the  hardware  world. 
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TRL  5  Software 


Definition:  Moduie  and/or  subsystem  vaiidation  in  a  reievant 
environment. 

Description:  Level  at  which  software  technology  is  ready  to 
start  integration  with  existing  systems.  The  prototype 
implementations  conform  to  target  environment/interfaces. 
Experiments  with  realistic  problems.  Simulated  interfaces  to 
existing  systems.  System  software  architecture  established. 
Algorithms  run  on  a  processor(s)  with  characteristics 
expected  in  the  operational  environment. 

Supporting  Information:  System  architecture 
diagram  around  technology  element  with  critical 
performance  requirements  defined.  Processor 
selection  analysis,  Simulation/Stimulation 
(Sim/Stim)  Laboratory  buildup  plan.  Software 
placed  under  configuration  management. 
COTS/GOTS  in  the  system  software  architecture 
are  identified. 
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A  relevant  environment  might  be  a  COTS  application  that  you  build  simulators 
around  to  represent  what  you  think  the  subsystem  will  look  like. 
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TRL  6  Software 

Minimum  Maturity  at  Milestone  B 

•  Definition:  Moduie  and/or  subsystem  validation  in  a 
relevant  end-to-end  environment. 

•  Description:  Level  at  which  the  engineering 
feasibility  of  a  software  technology  is 
demonstrated.  This  level  extends  to  laboratory 
prototype  implementations  on  full-scale  realistic 
problems  in  which  the  software  technology  is 
partially  integrated  with  existing  hardware/software 
systems. 

*  Supporting  Information:  Results  from  laboratory  testing  of  a 
prototype  package  that  is  near  the  desired  configuration  in  terms 
of  performance,  including  physical,  logical,  data,  and  security 
interfaces.  Comparisons  between  tested  environment  and 
operational  environment  analytically  understood.  Analysis  and 
test  measurements  quantifying  contribution  to  system-wide 
requirements  such  as  throughput,  scalability,  and  reliability. 
Analysis  of  human-computer  (user  environment)  begun.  47 


A  SW  business  system  is  often  not  designed  until  after  MS  B.  If  there  are  n 
different  competitors,  then  there  must  be  n  assessments.  To  achieve  this  TRL 
must  demonstrate  maturity  in  the  lab  that  is  doing  the  development,  i.e.,  must 
demonstrate  inhouse  maturity  (unless  there  is  a  subcontract). 

Another  possibility  is  to  perform  the  TRA  as  soon  as  possible  after  the 
Acquisition  Decision  Memorandum,  prefgerably  giving  the  program  only  limited 
authority  to  procedd  until  after  the  TRA  is  approved. 
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TRL  7  Software 

Minimum  Maturity  at  Milestone  C 

•  Definition:  System  prototype 
demonstration  in  an  operational  high- 
fideiity  environment. 

•  Description:  Level  at  which  the  program 
feasibility  of  a  software  technology  is 
demonstrated.  This  level  extends  to 
operational  environment  prototype 
impiementations  where  critical  technical 
risk  functionality  is  availabie  for 
demonstration  and  a  test  in  which  the 
software  technology  is  well  integrated  with 
operational  hardware/software  systems. 

•  Supporting  information:  Critical  technological  properties 
are  measured  against  requirements  in  a  simuiated 
operationai  environment. 
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MS  B  Requirement:  Demonstration  or  Validation 
in  a  Relevant  Environment  (TRL  6) 


A  relevant  environment  for  the 
demonstration  of  a  technology  is  a 
set  of  test  conditions  that  provide 
confidence  that  skillful  application 
of  that  technology  to  an  item 
(component,  subsystem,  or 
system)  will  support  the  required 
(threshold)  functionality  of  that 
item  across  the  full  spectrum  of 
required  operational  employments. 
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Example:  Software  componets  demonstrated  in  narrow  regime  (e.g.,  logistics). 
Need  to  have  arguments  and  tests  to  prove  that  it  will  work  in  other 
environments  as  well. 
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Demonstration  or  Validation  of  a 
Technology  in  a  Relevant  Environment 


•  Requires  successful  trial  testing  that 
either: 

-  shows  that  the  technology  satisfies 
functional  need  across  the  full  spectrum 
of  operational  employments,  or 

-  shows  that  the  technology  satisfies  the 
functional  need  for  some  important 
operational  employment  and  uses 
accepted  techniques  to  extend 
confidence  over  all  required  operational 
employments. 
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Need  to  have  a  trial  test  directly,  or  if  through  modeling  and  simulation,  need 
enough  data  to  support  that  it  applies  across  all  employments. 
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TRA  Performed 


•  Program  responsible  for  funding,  BUT 
most  of  the  work  has  already  been  done 

•  Independent  team  trained  and  convened 
by  the  Component  S&T  Executive  to 

-  Make  the  assessments 

-  Write  the  report 

•  Multiple  TRAs  should  be  conducted  if 
multiple  systems  still  in  competition 

Contact  DUSD(S&T)  with  any  issues  (e.g.,  CTE  uncertainty) 
early  in  the  process 


51 

See  additional  hardware  examples  See  additional  software  examples  See  additional  manufacturing  examples 

in  Section  C.2  of  the  Deskbook  in  Section  C.3  of  the  Deskbook  in  Section  C.4  of  the  Deskbook 


The  earlier  DUSD(S&T)  involved,  the  better. 


The  Component  S&T  Executive  should  not  ask  the  PM  to  do  the  TRA.  “The 
student  should  not  grade  his  own  homework.”  This  does  not  meet  the 
independence  criterion  for  a  TRA.  The  PM  should  supply  the  data  for  the 
independent  review  team  assessment. 


51 


Best  Practices  for  a  Preliminary  TRA  at 
Milestone  A  (Only  Applies  to  Ships) 


Exampie 

•  Use  the  TRA  to  identify  areas  for  management 
focus 

-  Create  critical  technology  IPTs 

•  No  contract  award  yet 

-  Update  the  TRA  after  a  selection  decision 

•  No  TRL  requirements 

-  TRL  of  3  or  lower  implies  higher  technology  risk 

-  Technology  Development  Phase  generally  mature 
technology  from  4  to  6 

-  Use  Technology  Transition  Agreements 
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Non  Complex  Technology  Is  Unacceptable 
Rationale  for  TRL  6  or  Higher 


Example 

•  Sub-system  identified  as  CTE 

-  There  is  not  a  similar  or  existing  prototype  that 
has  demonstrated  an  ability  to  perform  the  sub¬ 
system’s  mission  from  the  example  platform 

•  Inappropriately  assessed  at  TRL  6 

-  Although  the  sub  system  is  in  concept  design, 
its  low  technical  complexity  will  allow  use  of 
known  and  proven  fabrication  methods  and 
materials 


Demonstration  of  a  prototype  in  a  relevant  environment  is 
pre-condition  for  TRL  6 


Just  because  a  technology  looks  easy  to  implement,  you  still  must  demonstrate 
it  in  a  relevant  environment  for  it  to  be  TRL  6. 
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System  Level  Demonstration  Required  for 
TRL  7  or  Higher 


Software  Intensive  System  Example 

1  nappropriately  identified  two  CTEs  as  TRL  7  and  two 
as  TRL  8 

•  The  rationale  for  the  TRL  scores  is  that  the  systems  being 
scored  are  currently  in  operational  use  and  have  already 
been  through  the  acquisition  process.  Integration  into  a 
common  environment  is  the  major  area  to  be  addressed  for 
each  critical  technology.  The  panels  approached  the 
integration  issue  from  the  standpoint  that  integration  will 
occur  during  System  Development  and  Demonstration 
(SDD),  and  therefore  the  TRL  score  is  based  on  the 
individual  critical  technology. 

•  CTEs  should  have  been  assessed  to  be  TRL  6 
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Without  the  integration,  the  technologies  haven’t  been  demonstrated  in  an 
operational  environment. 
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TRL  7  Requires  Demonstration  of  a  System 
Across  Aii  Operationai  Environments 


Software  Intensive  System  Example 


•  TRA  stated 

-  For  the  purposes  of  this  TRA,  if  a  critical  technology 
has  been  implemented  in  the  operational  baseline  in  a 
limited  basis  (e.g.,  within  a  single  mission 
application),  it  is  considered  an  actual  system 
prototype  in  an  operational  environment. 

•  Many  CTEs  were  inappropriately  assessed  as  TRL  7 

•  TRL  7  requires  testing  across  all  potential 
operational  environments  or  basis  for  extending 
limited  tests  to  the  full  spectrum  of  operational 
environments 


Include  Sufficient  Data  in  the  TRA  to 
Evaluate  the  TRL  (hardware  1  of  3) 

Vehicle  Example 

•  The  Sub-System  X  was  identified  as  a  CTE 
for  a  Milestone  C  TRA 

•  No  specifics  were  provided  in  the  TRA  on 
how  Sub-System  X  met  its  requirements. 

The  TRA  only  included: 

-  Testing  underway ... 

-  Scope  of  test  is  ... 

-  Rounds  fired  to  date  are  ... 

-  Sub-System  X  has  been  incorporated  on 
other  platforms 

-  Integration  of  Sub-System  X  into  the  vehicle 
has  technical  challenges 

-  As  part  of  risk  mitigation  ... 


A  reviewer  must  point  out  areas  where  data  are  lacking. 


Include  Sufficient  Data  in  the  TRA  to 
Evaluate  the  TRL  (hardware  2  of  3) 


TRA  cover  letter  actually  provided  more 
specifics 

-  While  Sub-System  X  testing  has  shown  that  it 
meets  its  performance  requirements  the 
reliability  of  a  component  needs  improving.  The 
PM  has  identified  a  contractor  who  has 
considerable  experience  in  the  design, 
development  and  manufacture  of  these 
components.  Their  assessment  is  that 
incremental  improvements  in  design  of  Sub- 
System  X  are  possible  through  modest  design 
enhancements. 

TRA  7  was  assigned 
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TRA  preparer  must  include  data  used  and  references  for  those  data. 
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Include  Sufficient  Data  in  the  TRA  to 
Evaluate  the  TRL  (hardware  3  of  3) 


•  Observations 


-  The  TRA  should  identify  the  quantitative 
requirements  specific  to  the  CTE 


-  The  TRA  should  provide  the  resuits  of  tests 
reiative  to  those  requirements 

-  if  the  test  resuits  are  not  avaiiabie,  TRL  5  is  the 
maximum  achievabie 

-  The  TRL  7  in  this  example  was  inappropriate 


-  Miiestone  C  is  not  the  time  for  redesign 

-  A  Technology  Maturation  Plan  should  have 
been  requested 

•  Because  of  pressure  to  field,  the  ADM  authorized 
procurement  of  test  vehicies  and  long  iead  items  for 
additional  vehicles.  Further  procurement  was  contingent 
on  successful  demonstration  of  reliabiiity  growth.  58 


Include  Sufficient  Data  in  the  TRA  to 
Evaluate  the  TRL  (software) 

•  While  a  lot  of  work  went  into  the  TRA,  it  was  very  soft 

on  showing  quantitative  requirements  for  the  CTEs 

•  Excerpts  from  the  DUSD(S&T)  evaluation  ietter 

-  COTS  products  are  generaiiy  rated  as  being  TRL  5 
if  they  have  similar  examples  of  use  in  industry 

-  TRL  ratings  of  6  or  higher  must  be  based  upon  actual 
experience  with  those  products  by  the  receiving 
organization(s)  in  an  environment  close  to  that  intended  for 
the  deployed  system 

-  The  TRA  does  not  provide  sufficient  detail  on  how  the 
selected  products  fulfill  functional  expectations,  and  how  the 
results  of  the  pilot  efforts  justify  the  proposed  TRL  ratings 

•  Two  critical  technologies  were  used  in  pilots  but  no  results  are 
cited 

•  For  several  technologies,  the  pilot  results  were  described  simply 
as  no  problems  noted  during  the  pilot  efforts 

•  Four  of  the  critical  technologies  were  not  piloted 


Beware  of  TRLs  Driven  by  Non-Technological 
Considerations  at  MS  C  (1  of  4) 

Ground  Vehicle  Example 
•  CTE  identified 

-  Detection  System  X,  a  passive  infrared  detection 
system  that  detects  the  presence  or  absence  of 
chemical  warfare  agents,  was  identified  as  a  CTE 

-  The  Detection  System  X  was  to  be 
provided  by  another  independent 
program 
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Beware  of  TRLs  Driven  by  Non-Technological 
Considerations  at  MS  C  (2  of  4) 


•  Background 

-  Initial  plans  called  for  the  use  of  Detection 
System  Y,  a  different  infrared  detection  system 
Because  of  obsolescence  issues,  the 
requirement  became  specified  as  “performs  as 
well  as  Detection  System  Y  or  better”  and 
Detection  System  X  was  selected  for  integration 

-  While  Detection  System  X  was  not  meeting  its 
own  requirements,  the  program  was  restructured 
such  that  Increment  I  addressed  the  example 
ground  vehicle  needs  only 
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Beware  of  TRLs  Driven  by  Non-Technological 
Considerations  at  MS  C  (3  of  4) 


•  Component  assessment 

-  Detection  System  X  is  not  meeting  aii  of  its  revised  requirements 

-  Detection  System  X  provides  better  operationai  capabiiity  than 
Detection  System  Y.  its  performance  is  better  than  not  having  it  and 
having  it  does  provide  a  miiitariiy  usefui  increment  of  capabiiity. 

-  Enough  Detection  System  Xs  have  been  built  to  equip  the  entire 
example  vehicle  fleet 

-  A  TRL  of  7  was  assigned  by  the  Component  S&T  Executive 

•  ADM 

-  The  ADM  directed  the  development  an  acquisition 
strategy  and  plan  to  upgrade  Detection  System  X 
for  integration  into  the  example  vehicle  and  to 
assess  changes  to  force  structure  and  tactics 
resulting  from  current  limited  sensor  performance 
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The  TRL  7  assessment  was  for  the  example  ground  vehicle  purposes  only. 
There  was  no  separate  TRA  for  Detection  System  X. 
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Beware  of  TRLs  Driven  by  Non-Technological 
Considerations  at  MS  C  (4  of  4) 


•  Reviewer  Considerations 

-  Detection  System  X  TRL  could  be  no  higher  than  6  relative 
to  the  example  vehicle  requirements 

*  It  was  not  demonstrated  in  an  operational  environment,  it  is 
unknown  whether  it  was  even  demonstrated  in  a  relevant 
environment 

-  But 

*  Detection  System  X  was  produced  and 
fielded 

*  Detection  System  X  provided  a  militarily 
useful  capability  increment 

-  The  pressure  /  decision  to  field  the 
already  built  units  in  effect  waived  the 
example  vehicle  requirements,  making  a 
TRL  7  technically  correct 
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1.0  Purpose  of  This  Document 
2.0  Program  Overview 

2.1  Program  Objective 

o 

2.2  Program  Description 

2.3  System  Description 

3.0  Technoiogy  Readiness  Assessment 

3.1  Process  Description 

3.2  Criticai  Technologies 

3.3  Assessment  of  Maturity 

3.3.1  First  CTE  or  Category  of  Technology 

3.3.2  Next  CTE  or  Category  of  Technology 

3.4  Summary  of  TRLs  by  Technology 
4.0  Conclusion 


TRA  is  a  technical  report  with  references 


The  assessment  itself  is  the  meat  of  the  document.  The  process  through  which 
the  assessment  is  made  should  be  a  very  small  portion.  This  is  one  of  the  few 
technical  documents  that  go  to  the  Milestone  Decision  Authority. 
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TRA  Section  3.1  Should  Identify 
Independent  Review  Team  Members 


Operational  Planning  System  Example 


The  independent  TRA  team  is  composed  of  system  engineers 
from  an  FFRDC,  and  the  University  of  Goodness.  The  four 
principals  leading  the  assessment  are: 


-  Mr.  X,  FFRDC  Corporation.  Mr.  X  has  26  years  of  computer  systems  experience, 
and  16  years  experience  in  the  environment.  He  holds  a  Bachelors  of  Arts  in 
Quantitative  Methods  from  the  University  of  X  and  a  Master  of  Science  in 
Information  Systems  from  the  School  of  Engineering,  University  X. 


-  Mr.  Y,  FFRDC  Corporation.  Mr.  Y  has  24  years  of  computer  systems  experience, 
and  9  years  experience  in  the  environment.  He  holds  a  Bachelors  of  Science  in 
Computer  Science  from  the  University  of  Y  and  a  Master  of  Science  in 
Management  Information  Systems  from  the  University  of  Y. 


-  Mr.  Z,  FFRDC  Corporation,  Mr.  Z  has  7  years  of  computer  systems  experience. 
He  holds  a  Bachelors  and  Master  of  Science  in  Electrical  Engineering  and 
Computer  Science  from  the  Institute  of  Z. 

-  Dr.  A,  University  of  Goodness,  Senior  Technology  Consultant,  College  of 
Information  Science  and  Technology. 
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The  TRA  should  provide  the  names  of  the  independent  review  team.  A  brief  bio 
should  be  sufficient  to  show  their  technical  qualifications. 
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TRAs  Should  Not  Include  Pleas  for  a 
Favorable  Programmatic  Decision  (1  of  2) 


Example  1 

-  as  part  of  a  risk  mitigation  plan,  eight 
prototype  vehicles  with  the  System  are 
undergoing  testing.  Based  on  the 
results  to  date,  the  System  is  considered 
mature  enough  to  enter  low  rate 
production.” 

Example  2 

-  “...the  maturity  of  the  critical 
technologies  along  with  the  associated 
risk  mitigation  approaches  support  entry 
into  SDD ...” 
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The  TRA  should  show  what  has  happened  to  date.  It  should  not  contain 
opinions  or  legal  arguments. 
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TRAs  Should  Not  Include  Pleas  for  a 
Favorable  Programmatic  Decision  (2  of  2) 

•  Example  3 

-  “The  Example  3  Product  Manager  identified  two 
critical  Technologies  for  the  readiness  of  the 
program  to  enter  the  Design  and  Development 
Contract.  It  is  the  opinion  of  the  Example  3 
Product  Office  that  these  two  critical 
technologies  have  matured  to  a  TRL  level 
sufficient  for  entry  into  the  SDD  contract.  These 
two  technologies  will  have  matured  to  a  TRL 
level  sufficient  to  enter  LRIP  far  ahead  of 
schedule.” 

-  “Evolution  of  a  system’s  T/R  module  offers  the 
best  available  alternative  for  this  example  to 
meet  T/R  module  requirements  in  SDD.  This 
effort  is  imperative  to  the  success  of  SDD  and  is 
true  ‘risk  reduction’  ” 
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It  is  not  appropriate  to  argue  that  risk  reduction  is  underway  is  a  basis  for  a 
higher  TRL. 
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Title  10  requirement  for  certification 


•  H.R.  1815  Sect  801  §2366a.  (signed  Jan  6,  2006): 

Major  defense  acquisition  programs:  certification  required 
before  Miiestone  B  or  Key  Decision  Point  B  approvai: 
(a)  CERTiFiCATiON.  A  major  defense  acquisition  program 
may  not  receive  Miiestone  B  approvai,  or  Key  Decision 
Point  B  approval  in  the  case  of  a  space  program,  until 
the  milestone  decision  authority  certifies  that  - 
(1)  the  technology  in  the  program  has  been  demonstrated  in 
a  relevant  environment; 

(7)  The  program  complies  with  all  relevant  policies, 

regulations  and  directives  of  the  Department  of  Defense 


Certification  Submitted  Quarterly  in  the  SARs 
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Component  TRA  Coordination 

Identified  CTEs  and  assessed  TRL 

-  Component  TRA  approval  is  an  agreement  on  its  accuracy  only 
Maturity  requirements: 

-  Subsystem  demonstrated  in  a  relevant  environment  (TRL  6)  for  MS 
B; 

-  Prototype  (TRL  7)  or  actual  system  for  manufacturing  CTEs  (TRL  8) 
demonstrated  in  an  operational  environment  for  MS  C 

options  if  a  technoJpf*’''^^eA''  \itur^^ 

-  Requ^^Kl4jelair>'-^^  \  until  all 

int4  .  !>::r review  and  submit>^ 


tect 


laturation  plan  with  the  TRA. 


icquisition  Executive  submits  the  TRA  to  DUSD( 
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Approval  of  the  accuracy  does  not  necessarily  signify  a  decision  to  go  ahead. 
The  Component  S&T  Executive  may  recommend  any  of  the  three  options  to  the 
Component  Acquisition  Executive. 


We  have  labeled  the  go  ahead  as  planned  with  immature  technologies  option  as 
the  “last  resort”  because  of  the  nature  of  the  SDD  process.  It  is  highly  schedule 
driven.  The  technology  maturation  process  is  event  driven,  as  is  the  systems 
engineering  process.  Inclusion  of  two  event  driven  processes  in  a  schedule 
driven  environment  is  cause  for  concern.  So,  in  our  opinion,  the  “last  resort”  is 
really  a  best  practice. 


Unfortunately,  this  last  resort  is  becoming  standard  procedure  in  practice.  There 
is  a  great  deal  of  pressure  to  initiate  programs  -  that  is  the  point  where  funding 
normally  increases  substantially.  Some  have  suggested  that  6.2  and  6.3 
funding  limitations  have  prevented  programs  from  reaching  TRL  6  before  MS  B. 
On  the  other  hand,  there  are  programs  that  remain  in  the  tech  base  for  years.  In 
some  cases,  an  influential  champion  makes  the  difference. 
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DUSD(S&T)  TRA  Review 


•  Results  of  initial  review 

-  Concur 

-  Request  revisions 

•  Results  of  final  review 

-  Concur 

-  Concur  with  reservations 

-  Perform  independent  technical  assessment 


DUSD(S&T)  informs  Milestone  Decision  Authority  of  the  results 
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There  is  an  iterative  process  between  the  initial  review  and  the  final  review, 
scope  and  nature  depend  on  the  specific  situation. 


DUSD(S&T)  reservations  typically  become  language  incorporated  into  the 
Acquisition  Decision  Memorandum.  Such  language  may  call  for  an  updated 
TRA,  a  special  technology  IPT,  etc. 


To  date,  there  have  been  no  independent  technical  assessments. 


DUSD(S&T)  Review  Should  Evaluate  More 
Than  TRLs  (1  of  2) 


•  The  TRA  itself  is  a  technicai  report  ...  but ... 

-  Some  judgment  required  to  assess  whether 
TRLs  have  been  properly  assigned 

-  Factoring  the  TRA  into  a  programmatic 
decision  requires  a  great  deal  more  Judgment 


Treat  the  TRA  as  an  opportunity  to  gain 
insight  on  how  the  technology  maturity 
affects  programmatic  risk 


-  Coordinate  with  the  review  of  the  Systems 
Engineering  Plan 


-  Use  this  insight  in  recommending  whether  a 
delayed  entry  into  the  next  phase  should  be 
considered 


These  insights  should  be  passed  on  to  the  Milestone  Decision  Authority  so  they 
may  be  incorporated  into  the  decision  process.  Perhaps  Acquisition  Decision 
Memorandum  language  can  allay  concerns. 
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DUSD(S&T)  Review  Should  Evaluate  More 
_ Than  TRLs  (2  of  2) _ 

An  Example  in  Hindsight 

The  TRA  was  a  well  prepared,  substantial  document 
TRA  excerpts: 

-  the  basic  characteristics  outlined  in  the  KPPs 
were  evaiuated  using  various  existing  and  proposed 
R&D  hardware  and  software  programs...” 

-  “...  there  was  no  system  and  subsystem  models,  or 
prototypes  available  to  be  evaluated...” 

-  “...none  of  the  R&D  hardware/software  used  In  this 
assessment  wiil  be  used  directiy  in  the  system 
prototype...” 

1 1  CTEs  identified 

-  1 0  assessed  to  be  TRL  5 

-  1  assessed  to  be  TRL  4 

Three  years  later,  the  program  is  in  trouble,  unable  to 
meet  requirements 
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Outline 


•  Introduction 

•  Overview  of  Technology  Considerations  During 
Systems  Acquisition 

•  The  TRA  Process 

-  Identifying  Critical  Technology  Elements  (CTEs) 

-  Assessing  CTE  Readiness 

•  Technology  Maturation 

•  References  and  Resources 
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This  section  of  the  briefing  focuses  on  what  happens  if  the  technology  is 
immature  at  Milestone  B  although  some  of  it  is  also  applicable  to  all  CTEs. 
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Technology  Maturation  Policy  is 
Unambiguous 


“The  project  shall  exit  Technology  Developip^t  when 
an  affordable  increment  of  militarily-u^p^^^i^^^^bility 
has  been  identified,  the  technoj9P*^'^^^3i^^^^^crement 
has  been  demonstrated  ina<r^^^?l-<rfnvi^^  and  a 

system  can  be  devejpi>^'^^3<<^Liction  within  a  short 
timeframe  (norijv='^^^3<nan  five  years);  or  when  the 

MDA  decir^’^^^S^^>^^imate  the  effort . A  Milestone  B 

decisi<s;v^>/<i^the  completion  of  Technology 
Developci^nt.”  (DoDI  5000.2,  paragraph  3.6.7) 
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The  policy  is  absolutely  clear.  A  Technology  Development  Phase  exit  criterion  - 
or  depending  on  how  you  look  at  it-  a  System  Development  and  Demonstration 
Phase  entry  criterion  is:  the  critical  technologies  must  have  been  demonstrated 
in  a  relevant  environment. 
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Technology  Maturation  Policy  (continued) 


“The  management  and 
mitigation  of  technology  risk, 
which  allows  less  costly  and 
less  time-consuming  systems 
development,  is  a  crucial  part  of 
overall  program  management 
and  is  especially  relevant  to 
meeting  cost  and  schedule 
goals. 


Objective  assessment  of  technology 
maturity  and  risk  shall  be  a  routine  aspect 
of  DoD  acquisition.  Technology  developed 
in  S&T  or  procured  from  industry  or  other 
sources  shall  have  been  demonstrated  in  a 
relevant  environment  or,  preferably,  in  an 
operational  environment  to  be  considered 
mature  enough  to  use  for  product 
development  in  systems  integration. 
Technology  readiness  assessments,  and 
where  necessary,  independent 
assessments,  shall  be  conducted. 

If  technology  is  not  mature,  the 
DoD  Component  shall  use 
alternative  technology  that  is 
mature  and  that  can  meet  the 
user’s  needs.”  (DoDI  5000.2, 
paragraph  3.7.2.2) 
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The  policy  goes  on  to  say  that: 


1.  Technology  risks  must  be  managed  and  mitigated  and 

2.  If  the  technology  is  not  mature  enough  (i.e.,  it  has  not  met  the  SDD 
entrance  criterion)  use  something  else  that  is  mature 
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Guidance  for  Immature  Technologies 


If  the  system  does  not  meet  pre-defined 
Technology  Readiness  Level  scores,  then  a 
Critical  Technology  Element  maturation  plan 
is  identified.  This  pian  explains  in  detail  how 
the  Technology  Readiness  Level  will  be 
reached  prior  to  the  next  miiestone  decision 
date  or  reievant  decision  point.”  (Defense 
Acquisition  Guidebook  Section  4.3.2.4.3. 
Technology  Readiness  Assessment  (TRA)) 


*  TRL  6  required  at  MS  B. 

*  TRL  7  required  at  MS  C;  TRL  8  for  manufacturing  CTEs. 
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The  Defense  Acquisition  Guidebook  acknowledges  that  there  may  be 
exceptions  and  recommends  that  there  be  a  GTE  maturation  plan  for 
immature  technologies. 


Two  key  points  to  make: 

1 .  This  is  a  very  important  best  practice  and  we  recommend  that  there 
be  a  GTE  maturation  plan  for  every  GTE,  regardless  of  its  maturity  at 
the  Milestone.  After  all,  these  technologies  are  critical,  they  effect 
program  risk,  and  they  will  eventually  have  to  mature  to  TRL  9. 

2.  There  is  a  debate  on  how  strictly  the  policy  should  be  enforced. 

•  A  parallel  can  be  drawn  to  the  Systems  Engineering  Plan 
(SEP)  required  at  all  milestones  by  at  USD(AT&L)  policy 
memo.  OSD  provides  detailed  comments  (usually  critical)  on 
these  plans.  An  explicit  threat  to  stop  programs  if  the  SEP  is 
not  adequate.  This  is  a  relatively  strict  enforcement  of  policy. 

•  The  other  side  of  the  debate  says  the  programs  know  what 
they  are  doing,  they’ll  get  the  technologies  into  the  system  on 
cost  and  on  schedule.  That  is  reminiscent  of  the  George  Strait 
song  “Ocean  Front  Property  in  Arizona” 
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Programs  Do  Not  Uniformly  Understand 
Technology  Maturation  Policy 


Example  System 
System  will  include  technologies 
undergoing  continuous  maturation  until 
the  Critical  Design  Review 
This  approach  is  consistent  with  the^ 
emerging  DoD  Acquisition  guidelines 
and  will  be  typical  of  future  programs  in 
which  substantial  gains  in  capability 
are  sought 

The  TRA  plays  a  key  role  in  the 
acquisition  process  by  documenting 
the  current  state  of  maturity  and  path 
forward  for  each  of  the  critical 
technologies  to  enable  the  success  of 
this  approach 

By  clearly  identifying  the  challenges 
and  risks  for  the  critical  technologies  at 
this  point  in  the  program,  overall 
program  risk  is  reduced  and  can  be 
managed  with  the  SDD  phase 


This  approoch 
is  not 

consistent  with 
bobi  5000.2 


TRA  helps  ensure 
success  by 
potentially 
preventing 
programs  from 
moving  forward 
with  immature 
technologies  ^ 

How  can 

identification  alone 
ensure  that  risks  can 
be  managed  or 
reduced? 


This  slide  lend  credence  to  the  strict  enforcement  side  of  the  debate.  All 
programs  do  not  have  a  good  handle  on  acquisition  policy. 


Further  evidence  of  this  can  be  seen  with  some  statistics  about  the  SEPs  from 
one  Service.  The  SEP  is  plan  for  the  program’s  technical  development. 

•40%  of  programs  did  not  define  requirements  below  those  stated  in  the 
ICD/CDD.  One  program  said  “we  don’t  have  any  requirements.” 

•73%  of  programs  didn’t  have  or  didn’t  know  their  processes 
•45%  of  programs  didn’t  know  their  risks 

•45%  of  programs  didn’t  have  entry  and  exit  criteria  for  design  reviews 
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Technology  Maturation  Documents 


•  Technology  Development  Strategy  /  Acquisition  Strategy 

-  Defines  process 

•  MOA/MOUs 

-  Articuiates  externai  dependencies  on  other  programs 

•  Technoiogy  Transition  Agreements 

-  Articuiates  externai  program  dependencies  on  technoiogy 
base  programs 

-  Estabiishes  specific  technologies,  technoiogy  demonstration 
events,  and  exit  criteria  for  the  program 

•  Technology  Maturation  Plans 

-  Defines  how  CTEs  wiil  be  matured  in 
conjunction  with  the  Integrated  Master 
Schedule 

•  Risk  Mitigation  Plans 

-  Identifies  risks  associated  with  CTEs  and 
actions  for  mitigating  them 


These  documents  have  a  bearing  on  technology  maturity 

•We  have  already  discussed  the  TDS.  It  is  applicable  at  MS  A.  The  TDS 
evolves  into  the  acquisition  strategy  at  MS  B,  where  it  incorporates  other 
subjects  e.g.  risk  management,  T&E,  systems  engineering,  etc. 

•MOAs  and  MOUs  mostly  occur  in  Technology  Development  Phase 
•TTAs  with  laboratories  usually  occur  during  Technology  Development 
•TMPs  are  done  hand  in  hand  with  the  TRAs 

•RMPs  link  closely  with  the  TMPs  and  become  the  basis  for  tracking  GTE 
maturity  in  the  systems  engineering  technical  reviews.  This  reinforces 
the  point  made  earlier  about  the  need  for  every  GTE  to  be  tracked  in  the 
risk  database. 
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Responsible  IT^hnology  Maturation  Plan 
^  Outline 

•  Title 

•  Statement  of  the  problem 

•  Describe  the  technology  and  its  maturity  status. 

•  Say  how  this  technology  would  be  used  in  the  system. 

•  Solution  options 

•  Benefits  of  using  the  preferred  technology 

•  Fail-back  options  and  the  consequences  of  each 

•  Maturation  program  plan  with  schedule 

•  Describe  key  activities  for  preferred  technology 

•  Describe  preparations  for  using  an  alternative 

•  Show  the  latest  time  for  choosing  an  alternative 

•  Specific  actions  to  be  taken  (what  will  be  done  and  by  whom) 

•  What  prototypes  or  EMDs  will  be  built. 

•  What  tests  will  be  run 

•  How  the  test  environment  relates  to  the  operational  environment 

•  What  threshold  performance  must  be  met 

•  What  TRL  will  be  achieved 

•  Status  of  funding  to  perform  this  technology  maturation 


Status  monitored  at  technical  reviews  and  OIPT  meetings. 
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These  are  the  best  practice  contents  for  a  TMP.  The  plan  should  be  thorough. 
Doing  this  should  not  be  a  burden.  If  you  can’t  document  the  process,  you  don’t 
have  a  process. 


Obviously  what  you  see  here  looks  like  a  risk  mitigation  plan.  The  reason  is  that 
the  risk  mitigation  plan  was  the  basis  for  developing  this  best  practice. 

Recognize  however  that  the  risk  mitigation  plan  covers  more  than  technology 
risks. 


Of  course,  the  PM  must  be  the  one  to  prepare  the  TMP.  It  is  a  best  practice  to 
have  it  reviewed  by  the  independent  review  team. 
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Outline 


•  Introduction 

•  Overview  of  Technology  Considerations  During 
Systems  Acquisition 

•  The  TRA  Process 

-  Identifying  Critical  Technology  Elements  (CTEs) 

-  Assessing  CTE  Readiness 

•  Technology  Maturation 

•  References  and  Resources 
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References  and  Resources 


•  Defense  Acquisition  Resource  Center 
http://akss.dau.mii/darc/darc.htmi 

-  DoD  Directive  5000.1  (DoDD  5000.1),  The  Defense  Acquisition 
System,  dated  May  12,  2003 

-  DoD  Instruction  5000.2  (DoDI  5000.2),  Operation  of  the  Defense 
Acquisition  System,  dated  May  12,  2003 

-  Defense  Acquisition  Guidebook 

•  TRA  Deskbook 

http://www.defenselink.mil/ddre/doc/tra_deskbook_2005.pdf 

•  DDR&E 

-  Mr.  Jack  Taylor  jack.taylor@osd.mil 

•  Institute  for  Defense  Analyses 

-  Dr.  Cynthia  Dion-Schwarz  cdion@ida.org 

-  Dr.  Jay  Mandelbaum  Jmandelb@ida.org 
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Iniiiiliib 
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Example  Questions  to  Help  Identify  CTEs  for 
an  Aircraft  (Aerodynamic  Configuration) 


•  Does  the  design  incorporate  a 
configuration  that  has  not  been  used  in 
flight? 

•  How  similar  is  the  configuration  to  that  of 
aircraft  that  are  successfui? 

•  Does  the  configuration  impose  limitations 
on  control  authority,  stabiiity,  structural 
rigidity,  or  strength? 

•  Is  stability  acceptabie  at  high  angies  of 
attack? 

•  Is  stability  and  controi  acceptable  during 
configuration  changes  in  flight? 
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Example  Questions  to  Help  Identify  CTEs  for 
a  Networked  Communication  System 


•  Do  the  requirements  for  throughput,  data  latency, 
security  or  reliability  imply  that  a  new  or  novel 
technology  is  required? 

•  Have  the  network  routers  been  used  before  within 
the  required  performance  envelope? 

•  Are  new  or  novel  media  access  control,  coding, 
or  routing  algorithms  needed? 

•  Is  the  multiplexing  schema  new? 

•  Is  the  topology  (logical  and  hardware)  new? 

•  Do  the  peak  and  average  data  rates  require 
hardware  or  algorithms  in  the  system? 


Example  Questions  to  Help  Identify  CTEs  for 
a  Manufacturing  Technology 


*  Has  the  manufacturing  technology  been  successfully 
integrated  into  a  product? 

•  Is  the  industrial  base  capable  of  design,  development, 
production,  maintenance  and  support,  and  disposal  of 
the  system? 

•  Is  the  intended  design  producible? 

*  Have  the  materials  been  characterized  in  a 
manufacturing  environment? 


Are  the  materials  available  to  meet  quantity  and 
schedule  demands? 

Are  the  design-to-cost  goals  achievable? 

Are  the  key  manufacturing  processes  characterized, 
capable,  and  controllable  with  respect  to  achieving 
the  performance  requirements? 


Attainment  of  Technology  Readiness  for 
Hardware  CTEs 


Accomplishment 

TRL  Supported 

1 

2 

3 

4 

5 

6 

7 

8 

9 

Discovery  of  physical  or  mathematical  principle 

X 

Characterization  of  the  principle 

X 

Application  envisioned  and  described 

X 

Concept  of  application  analyzed 

X 

Critical  functionality  empirically  confirmed 

X 

Proof  of  concept  demonstrated  in  laboratory 

X 

Scale-up  or  other  extension  as  needed  by  concept 

X 

X 

Breadboard  or  component  tested  in  laboratory 

X 

Producibility  and  cost  estimated 

X 

X 

Engineering  Development  Model  (EDM)  of  component  tested  in 
laboratory 

X 

EDM  of  component  tested  in  relevant  environment 

X 

Prototype  component  integrated  into  a  system  EDM 

X 

X 

System  EDM  tested  in  simulated  environment 

X 

System  tested  in  limited  field  experiments 

X 

X 

System  tested  in  relevant  environment 

X 

System  tested  in  operational  environment 

X 

Production  system  tested  in  operational  environment 

X 

Production  system  proven  in  mission  operations 

X 

Attainment  of  Technology  Readiness  for 
Software  CTEs 


Accomplishment 

TRL  Supported 

1 

2 

3 

4 

5 

6 

7 

8 

9 

Discovery  of  mathematical  principle  or  algorithm 

X 

Characterization  of  the  principle 

X 

Application  envisioned  and  described 

X 

Concept  of  application  analyzed 

X 

Critical  functionality  empirically  confirmed  and  implemented 
software 

X 

Proof  of  concept  demonstrated  in  simulation 

X 

Scale-up  or  other  extension  as  needed  by  concept 

X 

X 

Component  tested  in  simulation 

X 

Producibility  and  cost  estimated 

X 

X 

Software  component  tested  in  an  integration  laboratory 

X 

Software  component  tested  in  a  relevant  environment 

X 

Prototype  component  integrated  into  a  system  prototype 

X 

X 

System  tested  in  a  simulated  environment 

X 

System  tested  in  a  limited  field  experiments 

X 

X 

System  tested  in  a  relevant  environment 

X 

System  tested  in  an  operational  environment 

X 

Production  system  tested  in  an  operational  environment 

X 

Production  system  proven  in  mission  operations 

X 

IT  TRA  Challenges 


•  TRA  /  TRL  model  derived  for  hardware-oriented 
systems 

•  Increasing  number  of  defense  acquisitions  are 
software  intensive 

-  Few  hardware  or  software  elements  can  be  singled  out  as 
CTEs 

•  New  IT  issues  include 

-  Interfaces 

-  Throughput 

-  Scalability 

-  External  dependencies 

-  Process  reengineering 

-  Information  assurance 

•  Environment  /  architecture  plays  are  greater  role  sg 


Software  Intensive  Systems  Fall  Into  Five 
Broad  Areas 


•  Information  systems 

•  Business  systems 

•  Networked 
communications 
systems 

•  Mission  pianning 
systems 

•  Embedded  IT  in 
tactical  systems 
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Information  Systems  and  Business  Systems 


•  Challenges 

-  Large  COTS  applications 

-  Integration  with  legacy 
business  applications 


Recommendations 

Start  with  lists  of  COTS 
products 

•  Focus  on  critical  applications 
used  in  a  new  or  novel  way 

•  Use  pilot  experience  to  justify 
TRLs  6  and  above 

Include  integrating  technologies 
where  applicable 
Pay  attention  to  DoD-unique 
environments 

Address  system-level  issues 

•  Responsiveness,  scaleability, 
etc. 
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Networked  Communications  Systems 


•  Challenges 

-  Services  focus 

-  Consolidation  of  user  needs 
and  anticipated  growth 

-  Managing  standards 

-  Technology  rollover 

*  Ability  to  provide  services 

•  May  transcend  individual 
products 

•  Information  assurance 


Recommendations 

-  Start  with  technologies  that 
enabie  one  or  more  services 

-  Avoid  process  issues  except 
where  enabled  by  technology 

•  Roll-out,  configuration 
management 

-  Estabiish  TTAs  where  DoD 
needs  not  met  by  commercial 
technologies 

•  E.g.,  mobile  ad-hoc  network 
protocols 

-  Consider  market  capabilities 
as  weli  as  specific 
technologies 
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Mission  Planning  Systems 


•  Challenges 


Recommendations 


-  Reliance  on  external  data 
sources 

-  Mixed  COTS/GOTS 

-  Infrastructure  upkeep  and 
modernization 


Start  with  required 
functionality  and  supporting 
technologies 
Identify  critical  data 
dependencies  on  external 
programs 

Assess  ability  to  succeed 
based  upon  total  suite  of 
data  suppliers  /  users  and 
infrastructure,  not  just 
application  maturity 
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-  Technology  turnover 

-  Scaleability  and 
responsiveness 


Embedded  IT  in  Tactical  Systems 


•  Challenges 

-  Lots  of  developed  software 

-  Military-unique  environments 

•  Radiation  hardened,  shock  / 
vibration,  high  reliability 

-  Military-unique  functionality 

*  IT  as  an  enabler 


Recommendations 

-  Start  with  function  domains 
or  WBS 

-  IT  typically  not  a  CTE  except 
where  consolidated 
computing  requirements  are 
used 

-  For  COTS,  carefully  examine 
relevant  and  operational 
environment  success  when 
rating  technology  readiness 

-  Do  not  address  developer 
capabilities  in  assessing 
technology  maturity 
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Attainment  of  Technology  Readiness  for 
Manufacturing  CTEs  (1  of  2) 


Accomplishment 

TRL  supported 

4 

5 

6 

Emerging  breadboard  design  options  provide  insight  into  potential 
manufaeturing  problems  with  the  industrial  base  infrastrueture  (faeilities 
&  manpower),  materials,  methods,  and  measurement  (inspeetion  &  test 
equipment). 

X 

Breadboard  design  options  provide  insight  needed  to  validate 
eharaeteristies  and  potential  geometries. 

X 

Various  strategies  identified  to  mitigate  teehnieal  and  eost  risk. 

X 

Prototype  brass  board  design  has  aetual  eomponents,  subsystems  or 
systems  that  have  assoeiated  manufaeturing  proeesses,  materials  and 
methods. 

X 

Preliminary  assessment  of  manufaeturing  assembly  sequenees. 

X 

Industrial  base  infrastrueture  (faeilities  &  manpower)  eapabilities  along 
with  measuring  and  test  equipment  initially  evaluated. 

X 

Cost  aeeounted  for  on  high  risk  manufaeturing  areas  and  plans  developed 
to  mitigate  risk. 

X 

Appropriate  quality  levels  have  been  aehieved. 

X 

Quality  management  model  understood. 

X 

Attainment  of  Technology  Readiness  for 
Manufacturing  CTEs  (2  of  2) 


Accomplishment 

TRL  supported 

7 

8 

9 

Manufacturing  processes,  materials  and  assembly  methods  have  been 
developed  for  a  production  environment;  ideally  in  a  pre-production 
facility  or  better. 

X 

Design  maturing;  key  materials  and  proeess  eharaeteristies  have  been 
identified,  and  planning  is  taking  plaee  for  managing  (proeess  eontrol  as 
appropriate). 

X 

Detailed  manufaeturing  risk  assessment  eovering  industrial  base 
infrastrueture  (faeilities  and  manpower);  materials  (availability, 
produeibility  eharaeteristies);  methods  (mature  proeesses);  measurement 
(inspeetion  and  test  equipment);  and  eosts. 

X 

Quality  management  strueture  identified. 

X 

Appropriate  throughput  levels  have  been  aehieved. 

X 

Initial  goals  set  for  yields,  quality,  and  reliability. 

X 

Manufaeturing  proeesses,  materials  and  assembly  methods  demonstrated 
on  produetion  representative  artieles  with  no  known  signifieant 
manufaeturing  risk. 

X 

Yields,  quality  and  reliability  within  25%  of  goals. 

X 

Design  mature;  proeess  requirements  proven  and  validated. 

X 

Quality  management  struetures  in  plaee. 

X 

Manufaeturing  proeesses  effieient  and  aeeeptable  in  faetory  environment. 

X 

Design  produeible;  used  to  produee  produetion  artieles  for  lOT&E  and 
the  field. 

X 

Design  to  eost  goals  met. 

X 

